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5 A Multi-Agent System for Automated Image Analysis 
and Understanding 

Background of the Invention 

The present invention relates to the field of image analysis and more specifically to 
methods for automated image segmentation, analysis and understanding by means of 
10 a novel Multi- Agent system. / 

There currently exists a broad spectrum of image analysis technologies and methods 
that try to provide automated image analysis. However, major problems arise from 
these methods as they are most commonly based on simple spectral thresholding 
techniques or variations thereof. There have been many attempts at providing efficient 

15 automated image analysis methods, some of which have been more or less successful, 
such as multiscale analysis and deformable models. In recent attempts, there have 
been many developments in methods that combine already existing techniques. 
However, the main problem with the combination of existing methods consists in 
determining the optimal set of parameters and order in which the methods should be 

20 ,. applied. With the recently added requirement. of symbolic image interpretation in the 
industry as well as in academia, in addition to image segmentation, standard image 
analysis methods have even more shown to be inadequate, to efficiently address 
these issues, a new theoretical field known as Multi-Agent systems has been adopted 
and formalized by the inventors specifically for the problems at hand. Although the 

25 basic concepts in Multi-Agent systems are known in the art, the present invention 
brings forth completely novel methods therein for allowing robust and fully 
automated image segmentation, analysis and understanding. 

Patent 5,978,520 provides a method and system for interactively generating vector 
data, more specifically to draw vector lines, from drawing data by means of a basic 

30 multi-agent system. The provided multi-agent system is based on the concept of a 
supervisor agent that enforces actions upon the agents it controls when problems or 
contradiction arise. In contrast to the invention described in 5,978,520, the present 
invention provides an automated system and methods for the automatic extraction and 
quantification of objects in images as well as symbolic image interpretation by means 

35 of a multi-agent system composed of fully autonomous agents. 
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5 Summary of the Invention 

An object of the present invention is to provide an image analysis and segmentation 
method and apparatus ^thereof for performing automated image analysis in various 
_ fields of applications where extraction and quantification of objects in images is 
required and where precise and robust analyses are of primary importance, 

10 Another object of the present invention is to provide a multi-agent system that 
integrates novel concepts in order to achieve full automation and maximum precision 
in image analysis contexts where complex image structures are required to be 
analyzed. The herein described invention therefore provides a multi-agent system that 
integrates the concepts of multiscale-based agent cooperation and sensing, sub- 

15 primitive agent specializations, dynamic agent parameterization as well as 
incremental feedback 

Overall there is provided methods and apparatus for first inputting digital images, 
then automatically analyzing the digital images by means of a specialized and 
autonomous multi-agent system that is initialized and assigned to the considered 
20 images by a program, and finally for displaying to the user and for saving in 
repositories the results generated by the multi-agent system. 

Brief Description of the Drawings 

The invention will be described in conjunction with certain drawings which are for 
the purpose of illustrating the preferred and alternate embodiments of the invention 
25 only, and not for the purpose of limiting the same, and wherein: 

Figure 1 displays the basic agent initialization and assignment process. 

Figure 2 depicts the various agent components for an embodiment. 

Figure 3 displays the basic flow of an example agent initialization and assignment 
process for one embodiment 

30 Figure 4 displays the entire system's basic operational flow. 

Figure 5 depicts the concept of an agent sensing an image by means of its sensors. 
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5 Figure 6 depicts the basic steps involved in the agent-to-agent communication 
process by means of the Receiver/Transmitter component 

Figure 7 depicts the concept of multiscale agent specialization and cooperation. 

Figure 8 is a representation of the components involved in providing multiscale 
" sensors. 

1 0 Figures 9a - 9b represent the concept of multiresolution image sensing. 

Figures 10a - 10c depict the process of multiscale information gathering by an agent. 
Figure 11 is a drawing of a typical satellite image landscape. 

Figure 12a - 12b illustrates the extraction of water course ridges and the process of 
agent initialization on these ridges. 

15 Figure 13 illustrates the initialization of Border Agents and the path following by 
Ridge Agents. 

Figure 14a - 14b illustrate the bridge detection process. 

Figure 15a - 15b represents the semantic interpretation of water courses. 

Figure 16a - 16b illustrate the detection of entities within water courses. 

20 Figure 17 illustrates a basic configuration of an apparatus for analyzing images 
comprising N processing nodes. 

Referring numerals comprised in the figures are here forth mentioned in the detailed 
description within brackets such as: (2). 

Detailed Description 

25 Although it is not the objective of the present invention to extensively describe 
hardware infrastructures that may be more optimal than standard personal computers 
as support to the herein described multi-agent system, we however describe a 
distributed hardware configuration that may be used in the present context. 



3 



WO 2005/038714 " PCT/CA 2004/00 1854 

5 Distributed Systems 

A multi-agent system is inherently highly distributable; from the fact that individual 
or a group of agents can reside on separate machines and communicate with one 
another over a network connection. From this, various types of distributed 
architectures can be considered in the implementation of a multi-agent system. 
10 Considering applications such as high resolution satellite image analysis, where 
considerable processing power is required to process the data in reasonable time 
frame, a distributed hardware platform may be required. A typical configuration is 
that of two or more networked computers where each computer (node) processes 
information using a program, and where a program communicates with another 
15 program residing on another computer through a communication interface. Patent 
6,350,239 discloses a basic distributed software architecture for medical diagnostic 
sy stems, where distinct modules that perform well defined and separate functions are 
executed on separate computers. The invention disclosed in 6,350,239 does not 
however address the challenging problem of distributing a single "module" over a 
20 plurality of nodes. Contrastingly, the herein described invention provides means for 
:\h distributing a single "module" over a plurality of computers or processing nodes, 
where each node may be host to one or a plurality of software agents that each 
perform a subtask of the global task (module). Through communication and 
scheduling of events, the plurality of software agents will achieve the required end- 
25 result. An example multi-node apparatus is illustrated in Figure 17. Depending on the 
nature of the application, the distributed system can be either loosely or tightly 
coupled. In the former case, the system can be a cluster-based backbone, where each 
processing node is physically located near one another, in a rack mounted 
configuration for instance, and where each node is networked to a neighboring node 
30 using a high-speed network interface and cable (such as 1Gb fiber-optics 
interconnects). The use of high speed networking reduces the communication latency 
between any program and module required to exchange information. In the loosely 
coupled scenario, the backbone may comprise networked computers physically 
remote to one another between which information can be exchanged using a 
35 networking protocol such as TCP/IP. 
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5 Although the present invention is detailed with an embodiment using a single personal 
computer as computatibnal backbone, it is understood that it can be extended to a 
more sophisticated architecture such as a cluster system. 

Entire System 

The entire system comprises two main segments: A first segment that integrates basic 
10 hardware and software components that form the basic backbone on which the multi- 
agent system relates, and a second segment that is the multi-agent system itself which 
comprises a plurality of software agents and basic software components. 

Referring to Figure 1 , the first process involved in the system consists in inputting one 
or a plurality of images by means of an input device (2). The latter can be a standard 
15 input/output device for reading digital data from a repository, or from an imaging 
device, and can comprise a communication interface to read the data from remote 
repositories over a network such as the Internet Once the image(s) read by the device 
and stored in memory, the Multi-Agent initialization and assignment program is 
instantiated. The program instantiates one or a plurality of software agents and makes 

20 these software agent instances accessible in memory (4). Once the agents are 
instantiated, the program assigns the agents to the images that are required to be 
analyzed (8). In one embodiment, once the agents have been instantiated, they 
become entirely autonomous entities, making their own decisions and acting 
according to their goals and states (6). In one embodiment, the agents can be 

25 randomly assigned to the images, wherein the program positions each of the agents at 
a random coordinate within the image. In another embodiment, the agents are 
strategically positioned within the image according to a priori knowledge on the 
problem at hand and general image structures, as well as according to the agent's 
specialization. In an example embodiment, the initialization and assignment program 

30 invokes a pre-processing method for detecting local minima within the image as 
initial landmarks for positioning the agents. Referring to Figure 3, this initialization 
process consists in first detecting the minima within the image using the said pre- 
processing method (24). For each of the said detected minima, spatial coordinates are 
extracted by means of the said method, wherein the said coordinates serve as the 

35 initial positioning coordinates, and are saved in memory (26). The initiation and 
assignment program is then input the set of coordinates, for which it automatically 
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5 assigns the initial agents by instantiating each of the agents and registering the latter 
with their according pixel coordinate and value (28). From 1 this point, the assigned 
agents will be autonomous and can move within the image to inspect other image 
regions and associated image .pixels. 

Referring to Figure 4, the entire system's overall operational flow comprises four 
10 main steps: first inputting a digital image to the system (30), then initiating the Multi- 
Agent system (32), followed by the actual step of image analysis by means of the 
Multi-Agent system (34), and finally, displaying the segmented image and associated 
results (36). 

Multi-Agent System 

15 The current invention relates to a novel multi-agent system devised for the automated 
analysis and segmentation of digital images. 

Agent Architecture 

By definition a multi-agent system comprises a plurality of agents. The present multi- 
agent system comprises a plurality of complex agents that have the means to receive 
20 information, take decisions, communicate, and take action, in an autonomous manner. 
To achieve this, a specific agent architecture that comprises a plurality of components 
has been devised. In one embodiment the components are the following: 

-Sensor(s) 

- Processors) 
25 - Actuator(s) 

-Receiver/Transmitter 

- Neural Processor(s) 

- State machine (s) 

- Memory unit(s) 

30 These components provide all the necessary means for creating autonomous and 
minimally "intelligent" agents. Each of these agents operates by first receiving stimuli 
from the external environment. Referring to Figure 2, an agent first receives input 
from either its sensor component (10) or its receiver/transmitter component (16). The 
sensors provide the agent with means for sensing its physical environment, which is in 

35 the present invention one or a plurality of digital images. In this sense, the sensors 
provide the agent with a view of the digital image, wherein the said view is composed 
of processed or unprocessed pixel values that are within the range of .the agent's 
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5 sensors within the image. Referring to Figure 5, in one embodiment the agent has two 
circular sensors (38) that are positioned according to the agent's position within the 
image, as depicted by (40) which refers to an image region. Image objects are 
indicated by (42). When the agent senses the image using its sensors, it has an 
appreciation of the image content based on the pixel values, from 0 to 255 for 
10 instance, comprised within its sensors. Again in Figure 5, the agent's sensors overlay 
portions of image objects that may have different pixel values (44 and 46). At this 
point, the agent receives image information from its sensors that indicates it is on 
image objects. 

As opposed to receiving image information by means of sensors, the agent's 
15 Receiver/Transmitter component provides means for the agent to communicate with 
other agents and therefore receive and/or transmit valuable information. This 
communication can be for instance with objective of negotiating, collaborating, or 
perhaps simply update some information or knowledge cumulated by the entire multi- 
agent system. In one embodiment, when an agent initiates communication with 
20 another agent, the said agent (48) first queries an Agent Registry (50) that holds a list 
of agents that are instantiated in the system along with their associated key or ID (52). 
The agent then uses the key or ID to open a communication channel with the other 
agent (54). Finally, once the communication channel with the other agent is 
established, the agents are free to communicate (56). In another embodiment, the 
25 Receiver/Transmitter component is used to send information to or receive from a 
shared memory, where information can be gathered and accessed by all agents. This 
shared memory can be a database or a specific data structure residing in memory and 
made available to all agents. 

Once external stimuli are input by the agent, the following steps involve multiple 
30 components and require the agent to take action, and may also require the agent to 

make a decision before taking action. Referring to Figure 2, the information input by 
. the sensors (10) is channeled to both or either the Neural Processor (18) and Processor 

(12) components. On one hand, the Processor performs various type of processing 

operations on the input information, such as, without limitation, edge detection, 
35 contrast enhancement or computation of statistics. On the other hand, the Neural 

Processor component can perform classification tasks on the raw input information or 
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5 on information processed by the Processor. In one embodiment, the Neural Processor 
identifies whether the input information is the edge of an object in the image. In 
general, the Neural Processor can identify the information input to it by means of 
classification or can incrementally generate a new knowledge base in conjunction 
with the Memory Unit (22). This incremental aspect of the Neural Processor provides 
10 the agent with means for learning on its own and can therefore adapt itself to new 
. problems and new types of images. This can be achieved by first inputting the sensor 
information in the Neural Processor, then automatically classifying the input 
information and saving a descriptor of this information along with its associated class 
in the Memory Unit. In one embodiment, the Neural Processor is comprised of one or 
15 a plurality of Artificial Neural Networks, in which case the learning process further 
adapts the Networks' weights. In another embodiment, the Neural Processor 
comprises Self Organizing Networks such as Self Organizing Maps, Growing Cell 
Structures, or any variants thereof. The self organizing networks can provide powerful 
means for automatically classifying and characterizing the visual information input by 
20 the agent's sensors. 

An agent further comprises an Actuator component (14) and can also comprise a State 
Machine (20). On one hand, the Actuator is the component that controls the agent's 
process of taking action. It is a general component and can control the process of 
communicating, proper information dispatching, adapting sensors, or even the process 

25 of moving in the image and changing its content Specific behavior or decision 
making can be achieved in conjunction with the State Machine which serves as the 
Agent's pre-defined mental states. In one embodiment, the agent's Actuator will force 
. the adaptation of the Agent's sensors based on the output of the Processor, State 
Machine, Neural Processor, or any combination thereof. The sensor adaptation can 

30 comprise, without limitation, changing the size of the sensors so as to view more or 
less information at one instant within the image. Another adaptation can be to enforce 
the sensors to view the information at a specific scale within a multiscale view of the 
image. In another embodiment, the Agent will move in a specific direction in the 
image according to the Actuator's command based on the knowledge acquired by the 

35 other components. This process of moving an agent in the image can take the form: 
move to coordinate (x,y) and sense image using sensors at these new coordinates. In 
which case, the Agent autonomously moves in the image. As the agents' general 
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5 objectives are to analyze and segment images, the Actuator component is also 
responsible for the action of changing pixels within an image, as part of the image 
segmentation process. When an agent is in the mental state of "segmentation", as 
defined by its State Machine component (see below), the agent changes the value of 
the pixel on which it finds itself. The change in value can be of binary form (change 
10 any pixel to a 1 or a 0), or can also be of any other value. Each agent specialization 
may change the pixel values to their own predefined value, which results in 
associating these specific pixels to their own class. For instance, a water agent can 
change the values of pixels it identifies as "water* to a unique value of 255, where 
pixels segmented by other specialized agents will have values different than 255. In 
15 the present invention, the original input images are never directly modified, but 
instead copies of the images are created and used by the system. In such case, the 
segmented image that comprises modified pixel values will be done on a copied 
image, used by every agent in the system. 

The agent's State Machine component provides means for pre-programming desired 
20 behaviors and patterns upon the activation of a specific state. . For any given state the 
agent is in, if this particular state is coded in the State Machine component, then an 
action or a sequence of actions that is to be taken is directly determined from this 
component Upon definition of an action or set of actions, the State Machine will 
work in conjunction with the Actuator for the agent to execute the required actions. In 
25 specific embodiments the State Machine component is defined by Decision Trees and 
Finite State Machines. An example sub-sequence from the State Machine component 
could be as follows: 

U- . * 

| IF sensor input information not classified by Neural Processor 
30 | Adapt sensors 
| Revalidate input 

The sub-sequence above provides a simple view of how the component can be 
involved in the execution of actions when the agent is in a specific state. In this 
35 particular example, the agent's Neural Processor was not able to identify the nature of 
the information input by the sensors and therefore finds itself in an "Information not 
Classified" mental state. This state is encoded by the State Machine and defines the 
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5 various actions required to be taken by the agent when such a state is encountered. 
The example sub-sequence is with respect to the dynamic adaptation of the sensors, 
where when the information is not classified, the agent is required to change its 
sensors' configuration (adapt) and revalidate the information input by the later. In one 
embodiment, when the agent cannot readily identify the information as being part of 

10 an image object's edge, the sensors dynamically change the scale and resolution at 
which the information is considered, a method that may facilitate the identification 
process. Although the Example State Machine sub-sequence is of simple nature, it is 
understood that the complete decision tree or finite state machine may comprise a 
broad range of states and decision processes. Although not directly illustrated in 

15 Figure 2, the Receiver/Transmitter component may also be controlled in conjunction 
with the State Machine, where for instance a simple incoming communication may 
trigger a predefined sequence of actions as defined by the component. 

According to the herein described Agent Architecture, the present Multi-Agent 
system can be composed of specialized agents that make decisions and take actions 

20 specifically according to their specialization and in accordance with the inherent 
collaborative aspect of the system. From the previous description of the basic 
components that compose an agent, it is clear that an agent can integrate derived 
components (same components with modifications) as well as a plurality of other 
components. In one embodiment, each agent specialization has sensors that can differ 

25 in shape, sensitivity, specificity and behavior. For instance, an agent that is 
specialized for the detection of object contours can have Neural Processor and 
Sensors that are specialized for the detection of edges. In another embodiment, an 
agent can be specialized for the analysis of high noise images, where the agent will be 
adapted to perceive the images as a multiscale representation so as to be robust to the 

30 noise and artifacts. A plurality of agent specializations can be readily devised with the 
herein described agent architecture. It is therefore understood that the Invention is not 
limited to the above described embodiments of agent specialization. 

Multiscale Perception 

Based on the beforehand described agent architecture, an aspect of the present system 
35 is the multiscale-based agent cooperation and image perception. The concept of 
multiscale perception herein considered comprises two main aspects: 1) multiscale 



10 



WO 2005/038714 



PCT/CA2004/001854 



5 perception achieved through the cooperation of agents specialized . for distinct or 
multiple scale levels, and 2) multiscale perception achieved by means of multiscale 
and multiresolution agent sensors. In the first case, the general concept of scale can be 
applied to both semantic and spatial scales, wherein the second case only applies to 
spatial scale. 

10 Cooperative Multiscale Perception 

In this first case, a number of agents are specialized and adapted to analyze the image 
represented at certain scale levels, where for instance agents specialized for analyzing 
an image at level 'N' (58) cooperate with one another and further cooperate with 
agents specialized for a level 'N-F (60). In this context, agents analyzing an image at 

15 a scale level N can facilitate the analysis process by providing new information to 
agents at finer scale analyzing the base image data. As the agents at higher scale have 
a different and broader understanding on the nature of the object or region currently 
being analyzed, the provided information to the lower level agents will allow the latter 
to adapt their parameters and reorient their paths if required. The higher level agents 

20 can also enforce restrictions on the lower level agents, where the agents can be bound 
to a specific region for instance. This multiscale cooperation can also comprise 
mutual feedback mechanisms where the knowledge and information gathered by 
agents at higher level can facilitate the lower level analysis, and where the lower level 
information can improve and facilitate the higher scale analysis. This coevolution 

25 provides robust image analysis means as it simultaneously takes into account multiple 
scale representations of an image, allowing for increased robustness to noise and 
providing a global and local view of the information. Figure 7 depicts this concept of 
multiscale cooperation where agents on multiple scales cooperate with one another in 
order to efficiently analyze the raw image data. The herein described method of 

30 cooperative multiscale perception and image analysis can further comprise the 
processes of agent reproduction and agent cloning where for instance when an agent 
that analyzes the raw image data finds itself in an uncertain environment or is unsure 
on the action it should perform due to the degenerated information, the agent has the 
possibility to clone or reproduce itself at higher scales, or inversely from higher scale 

35 to lower scale. This requires the agent to first identify the scales at which it will 
instantiate new agents, then clone or reproduce itself to these scales, allow the new 



11 



WO 2005/038714 



PCT/CA2004/0018S4 



5 agents to analyze the image they find themselves on, and update itself according to 
the information and knowledge provided by the new agents. 

Multiscale Sensor Perception 

In the case of sensor based multiscale perception, where only spatial scale is 
considered, the agents are composed of sensors adapted to provide a multiscale 
10 representation of the portion of the image being sensed. The agent therefore has the 
possibility of considering the image at one spatial scale level or at a plurality of levels 
within the multiscale representation. Referring to Figure 8, this mechanism comprises 
the steps of first reading the data that is in range of the sensors by means of the 
sensors(62), following this, the data is then input to the Processor component which 

15 computes a multiscale representation of the said data (64) using a method known in 
the art. The multiscale data representation can be input to the Neural Processor or 
further used by the Processor component In one embodiment, the agent considers 
only one specific scale level of the multiscale image representation. In another 
embodiment, the agent sequentially or simultaneously considers a plurality of scale 

20 levels. In yet another embodiment, the agent perceives the image at various spatial 
scales as well as in multiple resolutions, where the agent's sensors change in size to 
perceive a different level of information. This type of adaptation is usefid when an 
agent is required to further validate the environment in which it is found so as to 
increase or decrease its confidence of the structure it analyzes. Referring to Figure 9.a, 

25 the sensor data is input to either or both the Processor and the Neural Processor (66) 
where the data can be analyzed and processed, after which a response is sent to the 
Actuator that changes the size of the sensors (68) . according to the later response. This 
process is repeated as required. This sensor adaptation is represented in Figure 9.b, 
where the initial sensor state is depicted by (70), and where after adaptation the 

30 sensors are seen to be larger in size (72), meaning that more image data is being read 
by the sensors. In one embodiment, this type of adaptation is used to perceive a larger 
image region to ensure the agent is precisely detecting a contour. 

Agent Negotiation 

In order to resolve problems or contradictions as well as to facilitate the emergence of 
35 optimal solutions in the image segmentation process, the present system provides a 
means for Agent Negotiation. In a preferred embodiment, the agent negotiation is 



12 



WO 2005/038714 



PCT/CA2004/001854 



• 5 done by agents of different specialization that attempt to analyze the same image 
region. The objective is to allocate the image region to the agent that has maximum 
confidence on the segmented region. The process consists in first identifying the 
litigious region and for each agent to thereafter: 

- Evaluate considered region according to specialization; 
10 - Communicate with other agents in litigious state; 

- Vote for winning agent 

The first step of identifying the litigious region is achieved in one embodiment by a 
program that verifies if there is a plurality of agents that attempt to analyze a same 
image region. In another embodiment, an agent itself verifies if any other agents are 

15 analyzing the current region. In such case, the current agent evaluates the confidence 
that the said region is to be analyzed by itself. Following this evaluation, the agent 
communicates with the other agents in the litigious region and asks for their 
confidence value. The communicating agent can thereafter award the said region to 
the agent with maximum confidence. In one embodiment, the confidence value is 

20 computed by the agent's Processor. In another embodiment, the confidence value is 
attributed by a classifier such as a neural network using the agent's Neural Processor, 
where the neural network's activation value can be considered to weigh the 
confidence value. 

Agent Adaptation, Collaboration and Reproduction 

25 To provide increased robustness and intelligence to the system, the devised agents 
within the present multi-agent system have been incorporated means to dynamically 
adapt themselves to any image they analyze. 

A first type of adaptation is the context based agent adaptation, where an agent adapts 
its parameters and goals based on the information the agent gathers at jsl higher 

30 semantic or spatial scale or based on the information provided by another agent found 
at a higher semantic or spatial scale. The agent first begins analyzing the image at its 
current scale level, then, to validate that the segmentation is precise and correct the 
agent gathers information from a higher scale which provides detail on the location 
and nature of the object being segmented. By having knowledge on the nature of the 

35 object, as provided from a higher scale analysis, the agent's behavior and actions can 
be dynamically modulated and adapted to improve the current segmentation, if 
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5 required. As stated above, the gathering of information at a higher level can be 
achieved by the considered agent or by another agent at this higher level. In the 
embodiment of gathering higher level information, the process involves the steps of: 
1) Determining the scale level at which additional information should be gathered, 2) 
Proper agent initialization, 3) Information gathering, and 4) Agent adaptation. In a 
10 preferred embodiment, the adequate scale level is determined before-hand. The 
second step of agent initialization can take many forms, such as, without limitation, 
the agent moving itself to the higher scale, the agent cloning itself to the higher scale, 
and instantiating new agents (can be of different specialization) at the higher scale by 
reproduction or other means. In the first case where the agent at scale level / (74) 
15 moves to the scale level n (76), the later analyzes the region of interest at the higher 
scale to gain new knowledge on the feature or object being analyzed. Referring to 
Figure lO.b, this process requires the agent to first identify the coordinate which it is 
currently at within the image (78), then load the higher scale image representation 
from memory (80), set it as the current image of interest (82), begin analyzing the 
20 image at the previously identified coordinate (84), and finally update its memory and 
adapt its parameters (86). In the other case, the agent clones itself at the higher level 
and continues the analysis through direct collaboration with its higher level clone(s). 
This is similar to the just above stated method, but differs with the aspect of 
collaboration. Instead of a sequential process where the agent gathers information 
25 then adapts itself, the cloning method provides a parallel and dynamic adaptation 
method where the lower level agent dynamically adapts itself from the feedback 
provided by its higher level clones. Referring to Figure 10.C, this method requires the 
agent to first instantiate a riew agent of same specialization (88), copy its mental state 
and memory content to the new agent (90) so that the later becomes a clone (copy), 
30 position the clone at the proper coordinate at higher scale (92), have the clone analyze 
the image (94) and communicate the gathered information to the lower level agent 
(96). In a preferred embodiment, an agent can simply invoke its "clone" class method 
to clone itself, wherein the said class method is preprogrammed to generate a copy of 
an instance of an agent. The other method provided for proper agent initialization is 
35 the instantiation of new agents at the higher scale. As stated above, this can be 
achieved by simple instantiation or reproduction. In such case, new agents of same or 
different specialization will be instantiated and positioned at the proper coordinate on 
the higher scale image by the lower level agent. These new agents will thereafter 
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5 analyze the image region and provide new information to the lower level agent by 
means of communication. The overall process is identical to the previously defined 
method, wherein the only difference lies in the type of agents that are instantiated. In 
the context of agent instantiation by means of reproduction, it is possible to instantiate 
new agents of specific specialization depending on the reproduction process of a 

10 specific agent. In the present invention, agent reproduction can be general or 
specialized, where for instance in one embodiment an agent specialized to detect 
object contours has the possibility of reproducing itself in agents specialized in 
analyzing image background and object regions. By providing specialized 
reproduction schemes, agents have the possibility of collaborating and gaining new 

1 5 knowledge optimally and specifically for the tasks they are required to do. 

Another type of adaptation is based on reinforcement The agent begins a first 
analysis, and then after a given number of pixels have been analyzed, the agent 
reiterates the analysis from the start. This reiteration allows for the agent to iteratively 
increase its knowledge on the information that is being analyzed and also offers the 

20 possibility of exploring differing analysis methods. This method allows an agent to 
incrementally update its knowledge and understanding of the object or feature it is 
currently analyzing and can therefore optimally adapt its analysis parameters in 
accordance to its current knowledge. The adaptation involved can be with respect to 
the parameters relating to the Processor component or the Sensors, where for instance 

25 the agent can modulate its noise sensitivity, adjust its threshold parameters, and vary 
the scale and resolution of its sensors. 

Detailed description of the Invention in the preferred embodiment 
of Satellite Image Analysis 

A detailed description of the present invention will be further described hereunder 
30 using the example of detecting entities such as industrial buildings and bridges in 
satellite images, as well as further providing a symbolic interpretation of the image 
landscape. 

Firstly, the system reads a digital satellite image by means of its input device, as 
depicted by numeral 2 in Figure 1. An illustration of a satellite image is depicted by 
35 Figure 11, which contains bridges, buildings, roads and water courses. Then, the 
Program (4) preprocesses the digital image with objective of extracting the 
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5 . predominant water structures within the image which will thereafter serve as the 
initialization point for the multi-agent system. A possible method for extracting the 
main structures is to simply smooth the image at a high level by convolution with a 
Gaussian kernel followed by a threshold of pixels so as to only keep the pixels that are 
part of main water structures. Then the ridges of these main structures are extracted 
10 by means of a ridge extraction operator. Figure 12.a depicts the extraction of the most 
predominant water courses along with their ridges. Referring to Figure 12.b, 
following these preprocessing operations, the Program instantiates three agents 
specialized for the following of ridges, in accordance to the three ridge entrance 
points (98,100,102). The Program extracts the coordinates of these entrance points 
15 and assigns the agents respectively . at these coordinates within the image. The 
objective of the ridge agents is to follow the ridge that serves as a path and thereafter 
collaborate with other agents if required. Once these agents have been initialized and 
assigned, the multi-agent system becomes autonomous. In the next step, the ridge 
agents initialize border agents that will analyze the border of the water structures. The 
20 border agents are not directly guided by a path but instead use their sensors to orient 
their selves in the proper direction. The border agents are specialized to respond to 
bimodal input from their sensors as they are expected to perceive pixel values within a 
specific range on one side and pixel values with a different intensity range on the 
other side. This is from the fact that pixels that are part of the water structures will 
25 have a different intensity then the land or other non-water structures in the image. 
Based on this a priori knowledge, the border agent will be able to recognize water 
borders and move within the image in accordance, while at the same time segmenting 
the borders for thereafter allowing further analysis. 

The next step consists in the agents to move within the image and analyze its content. 

30 Referring to Figure 13, during the ridge path following, the ridge agents will 
encounter a forking in the path (104,106,108). When a ridge agent identifies the 
problem, it generates offspring of same type in a number according to the number of 
branching paths. These offspring will continue to follow ridges in an autonomous 
manner. At the same time, border agents analyze the image using their sensors and 

35 move along the water course borders. Referring to figures 14.a and 14,b, the border 
agents will encounter a bridge (1 12) in their path which will be seen, by the agent as a 
path discontinuity since the bridge line traverses the water and part of the bordering 
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5 land (1 14). This discontinuity will be identified by the concerned agent (1 10) and will 
trigger that adaptation of the agent's sensors to verify the nature of this discontinuity. 
The agent's sensors will successively increase in size and scale representation until 
the border edge on the other side of the yet unknown discontinuity (116) is sensed by 
the agent. At this point, the agent reproduces itself in a new Line Agent that is 

10 initialized on the line discontinuity. The objective is to allow this new agent to gather 
more information on the latter. At this point, the Border agent has identified a 
discontinuity that crosses the border which provides an insight on the nature of the 
latter: can possibly be a building, a port, a road or a bridge. The new line agent will 
provide the border agent with enough information to make the proper identification of 

15 the structure. Now that the line agent is initialized it begins the analysis and 
following of the line structure. This specific agent performs basic edge enhancement 
to compute the necessary information to move in the proper path direction and 
simultaneously analyses the pixel values on both sides of the line (118). By detecting 
and analyzing these pixels, the agent can verify whether the pixels correspond to 

20 water or land. Since in the present case the pixels along the path correspond to water, 
the line agent communicates with the Border agent and informs it that the 
discontinuity structure corresponds to the entity "bridge". The line agent continues its 
path until it reaches the end of the line, where if it encounters another border agent 
(120), it can also inform it on the nature of the current line structure. 

25 The next process in the analysis consists in providing a symbolic and semantic 
interpretation of the water courses. In such case, the agents are required to collaborate 
at various spatial scales in order to properly detect and understand the present 
structures, without being disrupted by noise,. artifacts, or other dominant structures. 
Relating to Figure 15, we can notice the presence of primary (122) and secondary 

30 (124) water courses that are to be detected, segmented and interpreted by the system. 
First, in continuation with the previous processes, we see that the ridge agents come to 
a point where their paths meet (126-128, 130-132). This contradiction triggers 
negotiation amongst the considered agents. In this case, as the agents correctly follow 
the same path, the negotiation will result in a fusion of both considered agents (126 

35 and 128, 130 and 132). Referring to Figure 15.b, the resulting fused agents (134, 136) 
will contain the knowledge acquired throughout the analysis of both agents, which 
means that the agent will have in memory the path of the complete water course ridge. 
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5 This fusion results in the generation of the symbolic concept of "same" meaning that 
this path consists of a single water course. Now that there remains two fused ridge 
agents, another level of interpretation can be provided. Both agents (134 and 136) 
have the knowledge that there initially was a split in path and that the result was a 
fusion of agents since they were on a same path. From this information, the agents' 
10 State Machines will provide the conclusion that both agents may be part of a same 
water course, arid that the separation may be caused by an entity such as an Isle, 
platform, or ship, for instance. To validate this, both agents instantiate an agent 
specialized in detecting blob structures at high spatial scales. Referring to Figure 16a, 
this is achieved through collaboration by first selecting a high spatial scale 
15 representation of the current image and instantiating both new agents (138, 140) at the 
coordinates of their respective "parents" in the new image. Their goal is to detect a 
blob structure (142) in the vicinity of their initial coordinates. The two lower-scale 
parent agents restrict the possible displacements of the higher-scale agents by 
confining them within the range of both' agents (144), based on their knowledge that 
20 the structure is between them. Once initialized, the agents go about analyzing the 
region to detect and segment the "blob" structure. Further in the process, both blob 
agents will try to segment the same pixels, which indicates that they are detecting the 
same structure. From this, the lower-scale agents will be provided with the knowledge 
that this is in fact the same water course containing an entity. At this step, the system 
25 may chose to continue the blob detection in order to provide symbolic and semantic 
interpretation of this entity. In the present case, based on the shape and spectral 
intensity of the entity, the agents can conclude that it consists in an Isle or sand bank. 
In another scenario, this entity could be a ship, where in such case, the blob agent 
would detect that it is a ship based on the entity's specific shape by means of its state 
30 machine and memory unit. To achieve this identification, the agents are first required 
to precisely segment the structure and extract shape descriptors, such as maximum 
and minimum diameters, as well as spectral descriptors. By inputting this information 
in both or either their Neural Processor and State Machine, the agents will have the 
possibility of precisely identifying the analyzed structure and assign it a symbolic 
35 value, as mentioned previously. 

Throughout these processes the border agents go upon segmenting the water borders. 
Now that the ridge agents have identified a single water course, the border agents 

18 . 



WO 2005/038714 



PCT/CA2004/001854 



5. collaborate with the later to segment the whole water course. The objective is to relate 
both borders to the appropriate water course, to quantify and analyze the water course 
and to thereafter identify connecting courses. Quantifying and analyzing the water 
course requires the agents to simultaneously perform the analysis at multiple spatial 
scales so as to gather the most precise and robust information, allowing for further 
10 detecting structures within and on the periphery of the water course. First, one of the 
remaining ridge agents positions itself at the beginning of the ridge path and displaces 
itself at the same coordinate in a higher spatial scale representation. Its objective is to 
inform the agents to which ridge and water course they belong to and to instantiate 
Water Agents that will analyze the water course. From its higher scale perception of 
15 the structures, the ridge agent readily sees that the border agents directly belong to the 
same water course. As the ridge agent moves along the ridge path it instantiates water 
agents. These new agents are specialized to analyze water structures by means of their 
multiscale sensors. These agents' sensors simultaneously perceive the locally sensed 
pixels at multiple spatial scales. This multiscale view generates a multiscale 
20 "signature" that is directly analyzed by their Processor and Neural Processor 
components. The Neural Processor will classify the signatures according to these two 
classes: water and non-water. This simple binary classification allows the agent to 
take the appropriate actions upon the classification output. If the Neural Processor 
identifies a signature as non- water, then the agent may be in the presence of an entity 
25 such as a ship. In a specific embodiment, the water agent can reproduce itself in 
agents specialized to segment such entities. The water agents are constrained by the 
borders previously segmented by the border agents, which prevents them from 
segmenting and analyzing structures that may have similar signatures as water pixels. 
Furthermore, since the water agents sense the image at multiple scales, they are robust 
30 to noise present within the image. At the end of the process, the water agents will 
have segmented the complete water course and other entities will have been further 
segmented and analyzed by other agents. The collaboration of border agents and 
water agents also further allows for the segmentation and identification of bordering 
structures such as buildings. In this case, the buildings will be seen as small 
35 discontinuities by the border agents, while at the same time perceived as non-water 
entities by the water agents. The water agents can therefore instantiate other 
specialized agents to segment and analyze these bordering entities. At the same time, 
the border agents can constrain the analysis domain of these new agents. 
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5 Overall,, by letting evolve the previously described processes, the entire image and 
image structures will be precisely segmented, quantified and characterized. The 
agents will go upon achieving their goals and the overall autonomous collaboration 
will result in the complete quantification and understanding of the therein contained 
structures. The final step of the overall process consists in storing the segmented 
10 image as well as the symbolic interpretation of every identified structure in a 
repository such as a database. The assignment program can execute this task by 
accessing the information contained in memory and appropriately storing it in a 
database. This stored information can be used by other programs, such as data-mining 
programs and display program, and can be directly interpreted by users. 

15 While the invention has been described in connection with specific embodiments 
thereof, it will be understood that it is capable of further modifications and this 
application is intended to cover any variations, uses, or adaptations of the invention 
following, in general, the principles of the invention and including such departures 
from the present disclosure as come within known or customary practice within the 

20 art to which the invention pertains and as may be applied to the essential features 
hereinbefore set forth, and as follows in the scope of the appended claims. 
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